由于机房电源割接失误导致所有服务器全部挂掉，导致数据库故障，重启数据库一直报错，以下是解决方法希望可以帮到有同样问题的小伙伴。

故障原因：Greenplum 由于备master节点故障导致gp无法启动处理

1、已经配置了gpstandby节点，如果gpstandby故障是不会影响Greenplum正常运行，但是如果此时关闭master主节点则无法启动需要将原gpstandby节点移除。

1、启动master节点失败

[gpadmin@gpmaster ~]$ gpstart 20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Starting gpstart with args: 20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Gathering information and validating the environment… 20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Binary Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’ 20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Catalog Version: ‘201310150’ 20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Starting Master instance in admin mode 20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master… 20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Setting new master era 20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Master Started… 20200908:11:30:40:025343 gpstart:gpmaster:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 2 Command was: ‘ssh -o ‘StrictHostKeyChecking no’ gpstandby “. /usr/local/gpdb-5.0.0/greenplum_path.sh; $GPHOME/bin/pg_controldata /master/gpseg-1”’ rc=2, stdout=’’, stderr='pg_controldata: could not open file “/master/gpseg-1/global/pg_control” for reading: No such file or directory ’

2、移除问题gpstandby

[gpadmin@gpmaster ~]$ gpinitstandby -r gpstandby 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------ 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Warm master standby removal parameters 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------ 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master hostname = gpmaster 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master data directory = /master/gpseg-1 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master port = 5432 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master hostname = gpstandby 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master port = 5432 20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master data directory = /master/gpseg-1 Do you want to continue with deleting the standby master? Yy|Nn (default=N):

y 20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Removing standby master from catalog… 20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Database catalog updated successfully. 20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Removing filespace directories on standby master… 20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Successfully removed standby master

3、停止master节点 [gpadmin@gpmaster ~]$ gpstop 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Starting gpstop with args: 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Gathering information and validating the environment… 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master… 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Greenplum Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’ 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:--------------------------------------------- 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Master instance parameters 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:--------------------------------------------- 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Master Greenplum instance process active PID = 19445 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Database = template1 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Master port = 5432 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Master directory = /master/gpseg-1 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Shutdown mode = smart 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Timeout = 120 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Shutdown Master standby host = Off 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:--------------------------------------------- 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Segment instances that will be shutdown: 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:--------------------------------------------- 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Host Datadir Port Status 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata50/gpseg0 40000 u 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata/mirror50/gpseg0 41000 u 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata50/gpseg1 40000 u 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata/mirror50/gpseg1 41000 u 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata50/gpseg2 40000 u 20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata/mirror50/gpseg2 41000 u

Continue with Greenplum instance shutdown Yy|Nn (default=N):

y 20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-There are 0 connections to the database 20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=‘smart’ 20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Master host=gpmaster 20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Master segment instance directory=/master/gpseg-1 20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-Terminating processes for segment /master/gpseg-1 20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-No standby master host configured 20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait… 20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-0.00% of jobs completed 20200908:11:31:15:025413 gpstop:gpmaster:gpadmin-[INFO]:-100.00% of jobs completed 20200908:11:31:15:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait… 20200908:11:31:15:025413 gpstop:gpmaster:gpadmin-[INFO]:-0.00% of jobs completed 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-100.00% of jobs completed 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:----------------------------------------------------- 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:- Segments stopped successfully = 6 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:- Segments with errors during stop = 0 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:----------------------------------------------------- 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Successfully shutdown 6 of 6 segment instances 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-No leftover gpmmon process found 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Cleaning up leftover shared memory

4、重新启动master [gpadmin@gpmaster ~]$ gpstart 20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Starting gpstart with args: 20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Gathering information and validating the environment… 20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Binary Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’ 20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Catalog Version: ‘201310150’ 20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Starting Master instance in admin mode 20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master… 20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Setting new master era 20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master Started… 20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Shutting down master 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:--------------------------- 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master instance parameters 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:--------------------------- 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Database = template1 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master Port = 5432 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master directory = /master/gpseg-1 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Timeout = 600 seconds 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master standby = Off 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:--------------------------------------- 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Segment instances that will be started 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:--------------------------------------- 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- Host Datadir Port Role 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata50/gpseg0 40000 Primary 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata/mirror50/gpseg0 41000 Mirror 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata50/gpseg1 40000 Primary 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata/mirror50/gpseg1 41000 Mirror 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata50/gpseg2 40000 Primary 20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata/mirror50/gpseg2 41000 Mirror

Continue with Greenplum instance startup Yy|Nn (default=N):

y 20200908:11:31:39:025522 gpstart:gpmaster:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait… … 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-Process results… 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:----------------------------------------------------- 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- Successful segment starts = 6 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- Failed segment starts = 0 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:----------------------------------------------------- 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-Successfully started 6 of 6 segment instances 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:----------------------------------------------------- 20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-Starting Master instance gpmaster directory /master/gpseg-1 20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[INFO]:-Command pg_ctl reports Master gpmaster instance active 20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[WARNING]:-global name ‘testurl’ is not defined 20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[INFO]:-No standby master configured. skipping… 20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@gpmaster ~]$

5、重新恢复maststandby主节点 [gpadmin@gpmaster ~]$ gpstate -f 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -f 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’ 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 8.3.23 (Greenplum Database 5.0.0-alpha.0+dev.266.g923109b build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) compiled on Mar 28 2020 11:36:47’ 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master… 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-Standby master instance not configured 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-------------------------------------------------------------- 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:–pg_stat_replication 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-------------------------------------------------------------- 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-No entries found. 20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------

[gpadmin@gpmaster ~]$ gpinitstandby -s gpstandby 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Validating environment and parameters for standby initialization… 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Checking for filespace directory /master/gpseg-1 on gpstandby 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------ 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master initialization parameters 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------ 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master hostname = gpmaster 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master data directory = /master/gpseg-1 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master port = 5432 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master hostname = gpstandby 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master port = 5432 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master data directory = /master/gpseg-1 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum update system catalog = On 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------ 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:- Filespace locations 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------ 20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-pg_system -> /master/gpseg-1 Do you want to continue with standby master initialization? Yy|Nn (default=N):

y 20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby 20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[WARNING]:-Syncing of Greenplum Database extensions has failed. 20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[WARNING]:-Please run gppkg --clean after successful standby initialization. 20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Adding standby master to catalog… 20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Database catalog updated successfully. 20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Updating pg_hba.conf file… 20200908:11:37:08:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-pg_hba.conf files updated successfully. 20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Updating filespace flat files… 20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Filespace flat file updated successfully. 20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Starting standby master 20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Checking if standby master is running on host: gpstandby in directory: /master/gpseg-1 20200908:11:37:22:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files… 20200908:11:37:23:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully. 20200908:11:37:23:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Successfully created standby master on gpstandby

[gpadmin@gpmaster ~]$ gpstate -f 20200908:11:37:27:025887 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -f 20200908:11:37:28:025887 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’ 20200908:11:37:28:025887 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 8.3.23 (Greenplum Database 5.0.0-alpha.0+dev.266.g923109b build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) compiled on Mar 28 2017 20:09:54’ 20200908:11:37:28:025887 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master… 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:-Standby master details 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:----------------------- 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby address = gpstandby 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby data directory = /master/gpseg-1 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby port = 5432 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby PID = 2641 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby status = Standby host passive 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:-------------------------------------------------------------- 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–pg_stat_replication 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:-------------------------------------------------------------- 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–WAL Sender State: streaming 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Sync state: sync 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Sent Location: 0/24000000 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Flush Location: 0/24000000 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Replay Location: 0/24000000 20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------